‫Extracting directory structure, crawling, so mapping the application layout and its structure is another
‫very important task.
‫Although there are some single page applications, the applications you will test can generally consist
‫of multiple Web pages.
‫And by multiple, I mean a lot in these pages can be independent or they can be linked to one another.
‫There's actually no magic to get the structure unless you're the actual developer of the application
‫and it is hours and hours of work.
‫So the best way to extract the structure of the application is to visit every page and click every link
‫and fill every form.
‫Then observe all the yourselves so you can manually walk through the application and identify Web pages
‫from authenticated and unauthenticated users perspective.
‫So this whole process is called crawling or spidering the application.
‫Now, you might think and you would be right that it's not really possible to do everything manually.
‫This really is a time consuming process.
‫So, gosh darn it, wouldn't you just love it if somebody created a tool for crawling, and how about
‫some scripts for good measure?
‫Well, I'm glad you asked.
‫Because now let's go over to Cali and open up your browser and berp, I know if you're raised like me,
‫it's rude, but I'm going to ask you to do it again.
‫Open your browser and burp.
‫OK, so sorry.
‫Go to Dashboard and make sure that capturing is active.
‫Then disable master interception from this button.
‫OK, so Skatoff.
‫But in your browser, redirect the traffic to berp by enabling Foxe proxy.
‫So now Berp will passively intercept all traffic coming from Firefox's.
‫And what we're going to do here is visit pages and click links.
‫So so that way, berp will create a structure of the application from the clicked you URLs by intercepting
‫passively.
‫Right, so I'm going to click on some links here on the page.
‫And then log in.
‫Open up a few other pages and.
‫OK, that's enough to show you.
‫So, OK, look at the history in BIR.
‫So all the requestor here.
‫And so the aim here is to see a sort of a broad overview of the application layout.
‫So.
‫There are a few more things in this step to perform, but I want you to just have a look here.
‫I'm sure that you've heard about robots that he.
‫I know almost every website uses this file to allow or disallow directories to be called by bots, so
‫robots dot text is a file that uses a specification protocol called robot exclusion protocol.
‫Now, it's not really something that you need to consider, but you might want to have a look at a later.
‫But, yeah, displaying this file is very handy if you want to see the sensitive pages and directories
‫easily.
‫So what am I saying?
‫Yeah, go to the robotics file of Boab.
‫And here are some directories that are not allowed.
‫So I'm going to visit each of them.
‫Admin directory.
‫Documents.
‫And images and then wait, what's this password's directory?
‫Now go to berp.
‫Click Target tab.
‫Quick site map.
‫On the left pane, you can see the site structure.
‫And also, there are filter options above, you know, this is great, I hope you get excited, as I
‫do so by clicking show all resources like 6000 and images, they're all going to show up on the map
‫as well.
‫I think the commercial version of Berp will do this task automatically, but I really do want you to,
‫you know, get your hands dirty, as I was saying before.
‫I know it's not the most efficient way, but this is how you learn, so we need something a bit more
‫intrusive.
‫Now, I should think so, Kelly.
‫Linux has a number of tools for this job.
‫There, Buster is one of them.
‫So open up the terminal and simply type there, buster.
‫And again, we will come up.
‫There, Buster, is the directory brute force here for Web applications.
‫So now let's provide the target, you, Earl, which is HTP Collins, so says one nine two one six eight
‫two zero for that one three zero BW slash.
‫And here's a starting point for door buster, and it will automate the tedious tasks of cataloging the
‫pages within the application.
‫That sounds good, huh?
‫So it works by requesting a Web page.
‫Parsing through it for links and then sending requests to these new links until all the Web pages are
‫mapped.
‫So then let's increase the number of threads here to 20.
‫And then choose a list, Brouse.
‫Now, there are several lists in Durban or directory under word lists.
‫And I'm going to choose the medium directory list.
‫OK, and QuickStart.
‫Oh, and another good thing is to identify administrative and test pages, these pages can contain sensitive
‫information and provide entry points to perform attacks such as a brute force attack.
‫And it's also possible to see old and backup files in the directory structure.
‫Don't laugh, I've seen it many times in real world situations.
‫So if the old version of the application functions and has any vulnerabilities, bingo, you can own
‫the entire system.
‫Besides that, the folders and files belong to the application, so there may be meta files and folders
‫of the server software as well as the application framework.
‫And I mean, no one files and folders such as my admin and so forth.
‫So Durbar here also looks for this kind of stuff.
‫And here, as you can see, it detects Drupal.
‫My admin and Escorial light.
‫OK, so at this point, I'm going to stop this scan and you can take a report.
‫Full text report.
‫Browse for the location, Issei and.
‫Give it the name.
‫I'm just going to type B rap and generate report.
‫So now you can go to that folder.
‫And here is a report we saved.
‫So now you can analyze it.